Bun.write(file, file): wait for a non-blocking fd instead of rejecting with EAGAIN - #46858

Repro (Linux, bun 1.4.3-canary.1+367d939d9 and main 179664e, 3 of 3 runs each)

head -c 1048576 /dev/zero | tr '\0' 'x' > big.bin
cat > cat.mjs <<'EOF'
process.stdout.isTTY;
try {
  const n = await Bun.write(Bun.stdout, Bun.file("big.bin"));
  process.stderr.write("resolved " + n + "\n");
} catch (e) {
  process.stderr.write(`rejected ${e.code} (${e.syscall})\n`);
}
EOF
bun cat.mjs | wc -c

main prints rejected EAGAIN (sendfile) and 65536. This PR prints resolved 1048576 and 1048576.

Reach. No issue reports this: searches for EAGAIN sendfile, EAGAIN splice and EAGAIN "Bun.write" find 0 issues. It was found during work on #41685. It is deterministic for a file above 65536 bytes and a pipe.

Who sets the flag, and who clears it

• A use of process.stdout or process.stderr in the script or in a dependency, for example a read of isTTY.
• A Bun that ran before on the same pipe. Bun does not clear the flag at exit (process: restore O_NONBLOCK on inherited stdio at exit #33827 proposed that and is closed). ( bun -e "process.stdout.isTTY"; bun cat.mjs ) | wc -c fails in the second process on main.
• One socket that is fd 0 and fd 1, as inetd and socat EXEC: start a server. The use of process.stdout makes stdin non-blocking too.
• Bun.spawn makes its own end of an extra "pipe" non-blocking (proc.stdio[3]).
• A spawn clears the flag on the fds that become fd 0, 1 and 2 of the child (src/spawn_sys/spawn_process.rs:793, since process.stdout/stderr: clear O_NONBLOCK for spawned children, don't drop console output on EAGAIN #43868). A copy after a spawn with an inherited stdout resolves on main.

How the copy waits

• read_retrying and write_retrying (src/sys/lib.rs) make one read or write. On EAGAIN, wait_until_readable or wait_until_writable blocks the thread with no timeout, and the call repeats. If the wait fails, the helper returns the EAGAIN.
• The wait is poll. On macOS it is select$DARWIN_EXTSN$NOCANCEL, which has no FD_SETSIZE limit. write_all_to_stdio (process.stdout/stderr: clear O_NONBLOCK for spawned children, don't drop console output on EAGAIN #43868) calls the same wait in place of its own poll.
• read_write_loop_capped copies into an fd destination. NodeFS::copy_file_using_read_write_loop copies into a path destination. Both call the helpers for each read and write.
• do_copy_file_range: EAGAIN from sendfile or splice returns fallback_read_write(remain, unknown_size, &mut total_written), as ENOSYS and EXDEV do. The loop continues at the byte where the kernel copy stopped. copy_file_range has no such arm: it runs between two regular files.
• The node:fs loop also serves fs.copyFile, fs.cp and Bun Shell cp. Its 5 other call sites open both ends from paths after an ISREG check of the source. On Linux a path gives a new blocking description, so those calls cannot get EAGAIN. On macOS and FreeBSD a destination such as /dev/stdout shares the description of fd 1, so there the loop now waits where it failed with EAGAIN. That was not run.

Tests. Each test spawns a child that reports { ready } and then how the promise settled. A child in a pipeline also reports its exit status.

1. A file to a stdout pipe that fills during the copy. Linux: sendfile copies 65536 bytes, fails with EAGAIN, and the loop continues at that byte. The child measures its CPU time for 1 s while the copy waits.
2. A stdin pipe to a stdout pipe that fills during the copy. Linux: the same for splice.
3. Linux only: a stdin pipe with no data yet, after fd 1 became non-blocking. splice fails with EAGAIN at byte 0.
4. A file to a full stdout socket.
5. A FIFO source that the child opens with O_NONBLOCK on an fd above 1024. The child measures its CPU time while the copy waits for data. The writer closes and the copy resolves.
6. A full FIFO destination. A reader takes a part and leaves, and the copy rejects with EPIPE write.
7. Linux only: one socket is stdin and stdout of the child, and the child uses process.stdout. Bun.write(path, Bun.stdin) gets three pieces, each after the one before it is in the file, and resolves with all 12 bytes. The file at the path has 5 bytes and a known size. The second piece ends in the sized part of the node:fs loop, the third in the part after it.

On main all 7 fail: 1 of 1 run of a debug build and 15 of 15 runs of a release build. On this PR all 7 pass in 30 of 30 runs of a release build and in 4 of 4 runs of a debug build. They also pass in 2 of 2 runs with the environment of the ASAN lane.

Each part of the fix breaks a test when it is removed (debug builds):

• No EAGAIN arm in do_copy_file_range: tests 1 to 6 fail with EAGAIN sendfile or EAGAIN splice.
• write for write_retrying in read_write_loop_capped: tests 1, 2, 4 and 6 fail with EAGAIN write.
• read for read_retrying in read_write_loop_capped: test 5 fails with EAGAIN read.
• read for read_retrying in the node:fs loop: test 7 fails after hel (the sized part) or after hello (the part after it).
• A wait that returns at once: tests 1 and 5 report 880 and 932 ms of CPU in the 1 s window. A copy that waits uses 5 to 8 ms (release) or 58 to 70 ms (debug).
• The two write sites of the node:fs loop have no test. They can get EAGAIN only on macOS and FreeBSD, through a path that shares the description of fd 1.

Measurements (release builds, linux x64, main 179664e against this PR)

• Syscalls of copies that never get EAGAIN, 10 workloads, 3 runs each: 0 of 10 differ. poll and select calls: 0.
  • The workloads: bench/cat/bun.js to a file and to a pipe, and stdin to stdout. The same three with BUN_CONFIG_DISABLE_COPY_FILE_RANGE=1. bench/copyfile/bun.js with and without that variable. 200 calls of fs.copyFileSync. A file to Bun.file(3).
  • The counted calls: sendfile, splice, copy_file_range, read, poll, ppoll, select, pselect6, fcntl, ftruncate, fstat, openat. The write count changes by one between runs on both builds.
• Instructions that run in user space for one 64 KiB chunk, counted with gdb stepi. read_write_loop_capped: 47 to 77, with 11 to 13 conditional branches. The node:fs loop: 61 to 69, with 11 to 12. Most of the difference is a write that is now a call.
• do_copy_file_range with sendfile: 33 instructions and 5 conditional branches for each call, on both builds. The two copy_file_range instances have the same size. The sendfile and splice instances grow by 2 bytes or 0.
• size: text 107,557,538 to 107,558,818 bytes (+1,280).
• struct CopyFile, its JobContext impl and CopyFile::create have no hunk: no new allocation, queue entry or field.
• CPU time of the writer for a 1 GiB file to | cat > /dev/null after a use of process.stdout. Medians of 12 interleaved runs, with the range: this PR 370 ms (224 to 709). The first revision, which stays on sendfile: 266 ms (159 to 536). main with a blocking fd 1: 192 ms (148 to 414).
• A 4 MiB file to a full pipe that a reader then drains 64 KiB at a time. 1 sendfile call fails with EAGAIN. Then 64 read and 64 write calls copy. 42 write calls fail with EAGAIN, and there are 42 poll calls.
• An idle stdin pipe after the reader of stdout left, for 3 s: 1 splice call, then a read that blocks. 10 to 16 ms of CPU. The copy resolves with 0 when stdin ends.
• worker.terminate() while a copy in the worker waits for a reader that starts after 2 s. main with a blocking fd 1: it settles after 1.65 s. main with a non-blocking fd 1: after 0 ms, because the copy rejected. This PR with a non-blocking fd 1: after 1.68 s.
• process.exit() on the main thread does not wait: the process is gone after 336 ms (main with a blocking fd 1: 337 ms).
• fs.promises.stat behind 12 copies that wait (12 pool threads): it settles when the reader starts, on this PR and on main with a blocking fd 1.
• process.stdout.write("A".repeat(200000)), then a copy of a 100 KiB file of B:
  • A reader that runs at once, 20 runs: the file is inside the A run in 14 and after it in 6. main rejects 20 of 20, and its B bytes are inside the A run in 6.
  • A reader that starts after 0.3 s, 10 runs: inside in 6, after it in 4. main rejects 10 of 10 and writes no B.
  • The script waits for the callback of process.stdout.write first, 20 runs: in order 20 of 20. main rejects 20 of 20.
• Tools: a ptrace syscall counter, gdb, objdump, size, nm. The machine has no strace or perf.

Not run here. No macOS or FreeBSD machine was available. cargo check passes for all 12 targets. On macOS, tests 1, 2, 4, 5 and 6 run the select wait, and tests 1 and 5 put the CPU limit on it. Windows is not affected and not run: only POSIX code sets O_NONBLOCK on stdio, and CopyFileWindows is other code.

An idle source whose reader left. The first revision rejected with EPIPE at once, because sendfile looks at the pipe first. Now the copy is in read and waits for input, as cat does. EPIPE comes with the next byte, or the copy resolves when the source ends.

Considered. A park on the io thread, as WriteFile does: every copy pays a job-list entry, and the loops still need a wait for the fds that cannot park. A wait that keeps sendfile and splice (the first revision): it saves the CPU time above and needs a rule for the end to wait for.

Other paths of Bun.write with the same trigger. This PR changes only the file-to-file job.

• A string or a small buffer: after a partial write and EAGAIN the bytes are written again from byte 0 (Bun.write(Bun.stdout, ...) duplicates the first 8192 bytes after a partial pipe write #45663, Bun.write: after EAGAIN continue with the unwritten tail, the count and the open fd #44389).
• WriteFile::do_write repeats write with no wait when it does not know that the fd can block (Bun.write: poll instead of spin when a nonblocking stdout pipe returns EAGAIN #35953).
• Wait for input when a synchronous read of stdin gets EAGAIN #46192 has its own wait for stdin (wait_for_stdin). The one of the two PRs that lands second can call bun_sys::wait_until_readable(Fd::stdin()) in its place. io: wait for a named pipe's EOF with select(2) on macOS #40099 declares the same select$DARWIN_EXTSN$NOCANCEL, which is a one-line conflict for the second.
• Bun.write(Bun.stdout, ...) has its own writer and does not go behind the data that process.stdout holds. console: output goes behind what process.stdout/stderr have buffered, as in Node #37128 keeps that: its list says that Bun.write(Bun.stdout) has its own sink, as on main.

Still wrong after this PR

• A program that runs after Bun on the same pipe still gets EAGAIN, because Bun leaves the flag set. ( bun -e "process.stdout.isTTY"; cat big.bin ) | wc -c prints cat: write error after 65536 bytes.
• A socket with a pending error queue and a full send buffer. poll returns at once for it, so the copy repeats write on a full core until the peer reads: 1493 ms of CPU in 1500 ms. Then it completes. main rejects with EAGAIN. console.log on main has the same loop since process.stdout/stderr: clear O_NONBLOCK for spawned children, don't drop console output on EAGAIN #43868 (1472 ms in 1479 ms). Another process must set SO_ZEROCOPY or SO_TIMESTAMPING on the socket for this: Bun has no API for them. A guard needs a count of retries in the helpers, and no test on a lane can reach it.
• macOS: a source that is not a regular file with a path destination. fcopyfile refuses it with ENOTSUP before a loop runs (Writing stdin to a file errors with ENOTSUP #14054).
• macOS: a path such as /dev/stdout with a regular source goes through fcopyfile. Its loop in Apple's copyfile.c ends at a write that fails.
• Linux: a pipe source with a path destination gives Non-regular files aren't supported yet (Bun.write(file, Bun.stdin): splice a pipe source into any destination on Linux #36692).
• EINTR from sendfile, splice and copy_file_range still ends the copy (node:fs, Bun.write, fetch: retry EINTR in the bun_sys wrappers instead of surfacing it #45929).

History. #38498 (closed in favour of #37128) had the wait and most of these tests. #43868 made the console writer wait in poll on EAGAIN (write_all_to_stdio). #37128 had this fix in commit 90e0621: EAGAIN from sendfile and splice went to the read/write loop, and read_retrying and write_retrying waited in both loops. Commit 9033ecd cut that PR down to the console change. This PR has that shape again. It adds the select wait for macOS and leaves out the EINTR arm, which #45929 owns. Bun.write(Bun.stdout, Bun.file(path).stream()) is a workaround on main: it goes through the stream sink and the event loop.

Questions for @dylan-conway

• Is the wait inside NodeFS::copy_file_using_read_write_loop still the place that you want? If node_fs.rs must stay as it is, the wait can move to CopyFile. Its calls with stat_size 0 can use read_write_loop_capped.
• The copy waits on its pool thread with no timeout, where a blocking fd already waits inside sendfile. Remove libuv on Windows #42819 adds JobContext::waits and WorkPool::schedule_wait for a job that waits for the other end of a pipe. This copy can report waits for a pipe or socket when that lands. Run pool jobs that own their memory without a VM borrow so worker.terminate() does not wait on a blocked Bun.write(file, file) #38312 is about worker.terminate() and a copy that blocks.

Overlap. #41685, #46636 and #45929 edit other lines of do_copy_file_range. copy_file.rs merges with each of them without a conflict (git merge-file). The 3-argument fallback_read_write that #41685 and #46636 call is the same.